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Abstract: 
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We examined a novel formative assessment and instructional approach with 89 stu- 
dents in three electrical engineering classes in special computer-based discussion sec- 
tions. The technique involved students individually solving circuit problems online, with 
their real-time responses observed by the instructor. While exploratory, survey and 
interview responses from 26 students suggest the technique offers important instruc- 
tional and assessment advantages: Compared to typical discussion sessions, a large 
majority of respondents reported being more engaged, learning more, and interacting 
more with the instructor. Students reported the anonymous mode allowed them to ask 
“dumb” questions. The instructor was able to address student problems and questions 
immediately, and the amount of formative assessment information from the interaction 
far exceeded what was available in typical settings. 
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Introduction 

A cornerstone of undergraduate education reform is the shift in 
emphasis from teaching to student learning. A key effort in reform is 
promoting active learning. Active (or student-centered) learning refers to 
the set of practices and activities designed to promote students’ engage- 
ment with content. Examples include collaborative learning (e.g., learning 
communities, group activities), undergraduate research, writing and oral 
communication across the curriculum, integrative learning experiences 
(e.g., capstone courses), self-learning activities (e.g., encouraging students 
to reflect on their learning), and activities that require students to engage 
in cognitive processes heyond memorization (Boyer Commission, 1998; 
Chickering and Gamson, 1987; Education Commission of the States, 
1995; Ewell, 1998; National Research Council [NRC], 1995, 2003; National 
Science Board [NSB], 2003; Terezini and Pascarella, 1994; Tinto, 2002; 
Wulf, 1998). 

Underlying student-centered learning is formative assessment. 
Formative assessments provide information on what, how much, and 
how well students are learning, for the purpose of improving the quality 
of student learning. Formative assessments require close observations 
of the student learning process, are ongoing, capitalize on the expertise 
and experience of instructors using the assessments, and respond to 
particularities of individual students, instructors, and class settings 
(e.g., Angelo and Cross, 1993; Baker, 1998; Black et ah, 2003; Black and 
Wiliam, 1998; Brookhart, 1999; Cross, 1988, 1998; Ewell, 1998; Sadler, 
1989; Wiliam et ah, 2004). Using formative assessments can have a 
profound effect on instructors and their understanding of their students 
(e.g., Steadman, 1998), and there is some evidence that formative assess- 
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ments improve student achievement (Brookhart, 1999; Olmsted, 1991). 
The promise of formative assessment is if instructors have timely infor- 
mation on whether students are learning; then their instruction can he 
modulated with greater precision and accuracy, resulting in improved stu- 
dent learning. 

Despite repeated calls for change in undergraduate education and 
encouraging findings when implemented, most colleges and universities 
have heen slow to adopt the practices known to have real impact on stu- 
dent learning (Boyer Commission, 2001). Current instructional practices 
in the science and engineering fields continue to rely on the lecture. Chen 
(2002) reported that 83% of science and engineering instructors use the 
lecture as their primary method of instruction in at least one class. The 
problem with lectures is not that the format is inherently ineffective; it is 
just that other formats are likely to he more effective for more students 
(Terezini and Pascarella, 1994). 

Several researchers have documented the undergraduate science 
and engineering student experience as an impersonal, anonymous, and 
competitive journey where students passively learn facts as they march 
through a rigidly structured curriculum taught hy instructors who are 
horing, inaccessible, and intimidating (Kardash and Wallace, 2001; Lewis 
and Woodward, 1984; Seymour and Hewitt, 1997; Tobias and Raphael, 
1996). Seymour and Hewitt found that one of the most common reasons 
directly contributing to students’ decisions to switch out of the major 
was faculty pedagogy (e.g., poor teaching, help with problems, advising). 
Dissatisfaction with faculty pedagogy - more so than poor academic 
performance - led to student departure and remained a source of dis- 
satisfaction with students who continued in the field (Adelman, 1998; 
Seymour and Hewitt, 1997). With a student departure rate of 40% (NSB, 
2004), it is hard to envision how the demand for science and engineering 
graduates will be met. 

Like Terezini and Pascarella (1994), we think the main issue with 
lecture classes is not the format per se. Rather, the learning environment, 
in the absence of effective instructional supports, often leads to open-loop 
instruction: an instructor covering material with little or no a) interac- 
tion with students; or b) information on what students have learned or 
not learned. The main determinant of learning is the degree of cognitive 
processing of the to-be-learned content, regardless of instructional format 
(deWinstanley and Bjork, 2002; Mayer and Wittrock, 1996; Wittrock, 
1989). That is, students who engage the material - who commit attention 
and cognitive resources toward comprehending and understanding the 
content - will learn more than those students who do not. The learning 
environment is a powerful mediating variable that affects learning indi- 
rectly by promoting or inhibiting meaningful engagement. 
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Formative Assessment in Lecture Classes 

Typical large-class instruction is one of “dead reckoning”: Marching 
through the syllabus with little information on how well (or poorly) 
students are really doing and hoping for the hest. Classroom assessment 
techniques have heen developed to address this situation and specifically 
designed to minimize the data-processing load on the instructor. For 
example, the one-minute summary, muddiest point, and other techniques 
are easy to administer and summarize (Angelo and Cross, 1993). These 
assessments focus on asking students what was unclear to them during 
the lecture. Informative as they are, however, these assessments are indi- 
rect measures of learning as they do not ask students to perform on tasks 
that indicate understanding of content. 

Increasingly, researchers are turning to technology to provide forma- 
tive assessment information (e.g., Fitch, 2004; Griffioen et ah, 1999; 
Heath et ah, 2005; Kashy et ah, 2001; Lieherman et ah, 2001; Marks, 2002; 
Schaeffer et ah, 2001; Twigg, 1999). For example, Weh-hased systems have 
heen used to deliver homework and low-stakes quizzes. The basic idea 
is that online systems can provide rapid feedback and students benefit 
from immediate knowledge of results and potentially, the opportunity for 
multiple practice trials. The impact of these systems on course outcomes 
has ranged from none (Bonham et ah, 2003), some (Hall et ah, 2004; 
Mestre et ah, 2002; Pauli et ah, 1999), to large (Oakley, 1996; Woolf et ah, 
2000). The key difference among these systems appears to be the quality 
of the interaction. Oakley allowed scoring of assessments and multiple 
types of interactions with instructor and peers (e.g., posing questions and 
issues). Woolf et al. used an intelligent tutoring system, which presented 
leading questions to guide students’ interaction with the system. In con- 
trast, the other systems provided only information on whether students 
go the problem correct. It appears Oakley and Woolf et al. used feedback 
effectively - providing information on whether the student was correct 
or incorrect, and if incorrect, specific steps to take to improve perfor- 
mance (Black and Wiliam, 1998; Kluger and DeNisi, 1996; Sadler, 1989). 
Interestingly, while all of these systems were individualized, scalable, and 
focused on content and performance as measured by multiple-choice tests, 
only the Oakley system allowed students to ask questions of their instruc- 
tors, although this was not in real-time. The capability to gather students’ 
input on what they are not understanding can be very informative. 

Finally, some approaches are based on real-time interaction tech- 
nology such as personal response systems (i.e., clickers). These systems 
have been used to increase student interaction in large lecture classes 
(e.g., Dufresne et al., 1996; Fitch, 2004; Heath et al., 2005). Such sys- 
tems provide an efficient and simple way of gathering student responses. 
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While the newer response systems are easy to set up and use for instruc- 
tors and students, they generally accommodate only selected-response 
formats (e.g., multiple-choice) and offer little or no support for observing 
students’ problem solving or more complex communication modes (e.g., 
open-ended responses). The capability to observe the process of students’ 
problem solving is important, particularly in engineering when how one 
solves a problem is often more important than the final answer. 

Our conclusion is that the most important elements of an effec- 
tive formative assessment system are a) the capability of the system to 
provide good information about what students know and do not know; 
and b) the use of that information by instructors to provide feedback to 
students about their performance, and to adjust instruction accordingly. 
As with effective tutoring (Bloom, 1984) and instructional techniques 
that promote interaction and engagement (Hake, 1998), the fundamental 
enabling capability is the bidirectional flow of accurate information 
between student and instructor. 

Research Questions 

Thus, our study focused on capturing a student’s problem-solving 
response when it was “hot.” Our assumption was the accuracy and 
precision of the assessment information are highest when the students 
are engaged in the process of problem-solving. Our research was exploratory 
and focused on utility and feasibility: To what degree did students perceive 
that the system promoted learning, interaction, and engagement with the 
content? To what degree did the system provide the instructor with useful 
information about students’ understanding? 
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Method 

Participants 

The computer-based discussion sections were held during the instruc- 
tor’s office hours and participation was voluntary. We tested the system 
in three different undergraduate electrical engineering (EE) classes at a 
large public university in Southern California: EEIO (Circuit Analysis I), 
EE115A (Introduction to Analog Circuits 1), and BREES (Bridge Review 
for Enhancing Engineering Students). Five discussion sessions were held 
in EEIO and EE115A, and three sessions for BREES. The same instructor 
taught all courses. 

Across all courses, 89 students participated at least once. The sessions 
were typically attended by roughly 20 students. In general, out of all stu- 
dents who attended, over half attended two or more sessions. Of all the 
participants, 26 participants responded to requests to complete a survey 
at the end of the quarter. Of the 26 participants, 8 agreed to participate 
in interviews. Thus, the findings should be interpreted as exploratory. 
Survey respondents were paid $10 for their time, and interviewees were 
paid $20. 

Table 1 shows the participation rates by course. The table values show 
the number of times the same student attended the computer-based 
discussion section. For example, the computer-based discussion section 
was offered five times in EE115A, and 5 students attended all sections and 
8 attended two sections. In general, about half of the participants attended 
only one section. 

Table 1 : Number of Times a Student Attended the Computer-based 

Discussion Section 



Number of times the computer-based section was offered 

Course 

1 

2 

3 

4 

5 

EEIO 

14 

6 

8 

4 

N/A 

EEl 15A 

24 

8 

2 

9 

5 

BREES 

1 

0 

8 

N/A 

N/A 
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Online Formative Assessment and Instructional Tool 

The tool we used was Discourse (ETS, 2003). Discourse is a client- 
server formative assessment tool that provides instructors the capability 
to author and administer assessments, track student responses, and 
generate reports of student performances. For the purposes of this study, 
we used only its interactive problem-posting and discussion capabilities. 

Each student used a laptop, which was connected wirelessly to the 
instructor laptop. The instructor would post a prompt (e.g., “Compute the 
voltage at node 1.”) and circuit schematic. The student computer received 
the posting and displayed the prompt and circuit schematic. Students 
typed their responses in a text-entry area and could respond with an equa- 
tion or enter a question or other response. Figure 1 shows the student user 
interface. The box below “Answer” is where students typed their responses. 
While Discourse was designed with numerous formative assessment capa- 
bilities, we used only its instructional capabilities, which included the 
capability for the instructor to present a prompt and schematic and to 
observe students’ real-time responses. 

Figure 1 : Student User Interface 
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As shown in Figure 2, each student’s anonymous response (keystroke 
level) was displayed in a row on the instructor’s computer, allowing simul- 
taneous observation of all student responses (large window). Each student 
was identified by a number (“Name” column). Students could also commu- 
nicate via an anonymous instant messaging interface to ask the instructor 
questions and receive an individualized response (screen not shown, 
but essentially a “chat” application). The instructor could also use the 
whiteboard and conduct whole-group discussions, which were based on 
the real-time information from the students. 

Figure 2: Instructor User Interface 


Question; 

Write the node voltage equation at the node attached to the inverting input (Va - V-) / !2 + (Vb - 
V-)/1 20 + ( Vc - V-)/450 - V-/600 - (V- - V0)/1 80 = 0 


Info I 

OK 

Tlies 

Response 



2001.EE10- 

0 

1 

1 don't understand operational ^plifiers as of yet and was hoping to learn it today in office hours, 
txit 1 will venture a (wild) guess: 






(va • v-y72 (v- • vb)71 20 (v- • vc)/450 v-7600 (v- • Vo)71 80 = 0 



2002, EEIO- 

0 

1 

-V-/600->(Vo-v-)/1 80t(Va-V-)/72-f(VbV-)/1 20t(Vc-V-)7450 
1 must admit 1 have sINI to review this type of ptoblem. 



2003.EE10- 

0 

1 

V- ■ VC7450 (v- ■ vb 7 120) + (v- ■ va7 72) -f ( v- 7 600) (v- ■ Vo)7180 v- = 0 



2004, EEIO- 

0 

1 

[ Va • V- ]772 + |vb ■ V- ]71 20 [vc • v-]7450 ■ (v- ■ vo)71 80 • (v- 7 600) • 0 where v- - v+ if ideal • 
vd 


2005.EE10- 

0 

1 

(v 17450 ♦|v--Vo)71 80 -rv-7600 


I2006.EE10- 

0 

1 

(V- • Va)772 (V- ■ Vb)71 20 + (V- • Vc)7450 ♦ V-7600 - (V- ■ V0)71 80 


2007.EE10- 

0 

1 

v7600-f(v-Vo)7180+(v-vc)7450+[v-vb)7120+(v-va)772 = 0 


2008.EE10- 

0 

1 

(V- - Vc)7450k ■ [V- - Vb)71 20k ■ [V- - Va)7 72k - v-7600k + (Vo ■ V-)71 80k - 0 


'?nin FFin- 

n 

n 

ZA 
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Measures 

Data were gathered from student surveys, student interviews, and 
instructor interviews. 

Student Surveys 

The student surveys and interviews focused on gathering information 
on students’ perceived impact of the experience, relative to other typical 
discussion sessions. The survey mainly consisted of selected-response, 
Likert-scaled questions. In addition, participants were provided with space 
to write comments or specific examples. 

Learning of Course Material 

Participants were asked to indicate, on a 5-point Likert scale, how 
effective the computer-based discussion was compared to typical sections 
with respect to learning of course content {1-much more effective, 2=more 
effective, 3=similar, 4=less effective, 5-much less effective). 

Interaction 

Participants were asked to indicate, on a 5-point Likert scale, how 
effective the computer-based discussion was compared to typical sec- 
tions in terms of facilitating interaction with the TA or instructor {l=much 
more effective, 2-more effective, 3-similar, 4=less effective, 5=much less effec- 
tive). We defined interaction as communicating or asking questions, or 
responding to the TA or instructor. 

Engagement 

Participants were asked to indicate, on a 5-point Likert scale, how 
effective the computer-based discussion was compared to typical sections 
in terms of facilitating engagement with the material that was being cov- 
ered {l=much more effective, 2=more effective, 3=similar, 4-less effective, 
5-much less effective). We defined engagement as how much students were 
thinking about or working on the course material, as opposed to doing 
homework for other classes, playing games, and engaging in other non- 
content-related activities. 

Overall Impression 

This item was intended to gather participants’ overall impression 
of their experience. Participants were asked to indicate their impres- 
sion of the computer-based (i.e.. Discourse) sessions on a 4-point Likert 
scale {l=very positive, 2-moderately positive, 3-moderately negative, and 
4-negative). 
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Specific Course Outcomes 

Finally, we asked students to report on a 5-point Likert scale (l=very 
big effect, 2-big effect, 3-some effect, 4-little effect, 5-no effect, and not 
applicable), how much of an impact the computer-based section had on 
the following outcomes: 

• learning of specific analysis techniques 

• development of circuit analysis problem solving skills 

• completing homework assignments 

• performance on exams 

• performance on the course (EE10/EE115A/BREES) 

• the student’s improvement on attitudes toward electrical 
engineering 

• the student’s attitudes toward (EE10/EE115A/BREES) 
in particular 

• the student’s attitudes toward engineering in general 

• the student’s making new friends/networking 

Student and Instructor Interviews 

The student interviews were intended to gather in-depth responses 
to the issues of learning, interaction, and engagement. Interviewees were 
asked to elaborate on their survey responses. The course instructor and a 
TA were asked about their perceptions on the same issues, but from their 
perspective as instructors. Interviewees were free to elaborate on as many 
issues as they saw relevant. 


Results 

Our first research question. To what degree did students perceive that 
the system promoted learning, interaction, and engagement with the material? 
was addressed with student survey responses to questions, written com- 
ments, and follow-up interviews. The second research question. To what 
degree did the system provide unique information about students’ learning and 
understanding? was addressed with instructor interviews. We examined 
the relation between frequency of participation and specific course out- 
comes (homework, exams, and grades) and found no significant relations. 
This result should be interpreted in light of the voluntary and transient 
nature of students’ participation in the study, a situation that resulted in 
varying treatment lengths (Table 1, page 8). However, evidence from the 
student surveys and interviews was convergent and is presented next. 
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Students' Overall Impression 

In general, participants perceived the computer-based section posi- 
tively. Four rated the section as very positive, 16 as moderately positive, 
and 4 moderately negative. No one reported the experience as negative. 
While the student survey responses were encouraging, the follow-up 
interviews offered insight into the learning processes occurring during the 
computer-based section. 

One of the most discouraging findings was an echoing of Seymour and 
Hewitt’s (1997) findings of poor instruction in engineering in general. In 
most of the interviews conducted, students complained of the quality of 
instruction of most professors. Conversely, one of the most encouraging 
overall findings was students’ reporting of a dramatic shift from passive 
to active learning. In regular discussion sections, students reported being 
passive compared to the computer-based section, where they participated 
and engaged in problem solving. One student reflected on this change of 
roles: 

At first I was kind of confused about what we were doing. After every 
session, i beiieve it was two or three times, it was actuaiiy kind of heipfui. 
Because we did the exampies and we actuaiiy had to participate in ciass, 
or were expected to participate and do the exampies and give the answers 
and then check them. 

Students reported being very satisfied with the section because they 
could visualize circuit analysis and see it more accurately on the computer. 
When asked if the computer-based discussion section had any impact on 
the level of comprehending circuit analysis, one student responded: 

Biggest impact because we were actuaiiy using it so that's a big impact. 

You understand it better than before because we were using it. We were 
doing concrete exampies. i can see... if i make a mistake... You can weed 
those out by doing probiems there, then when you go to the exam, you 
pretty much know how to do the probiems, because you've aiready done 
it before. 
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Students' Perception of Learning, Interaction, 
and Engagement 

As shown in Table 2, when participants were asked to compare the com- 
puter-based section to typical discussion sections, the general response 
was positive. The computer-based section appeared to have the largest 
impact on students’ engagement with the material. In any case, student 
responses were generally positive across all three areas, with only two or 
three students reporting their experience in the computer-based section 
to be less effective than typical sections. 


Table 2: Students' Perception of the Relative Impact of the Discourse 

Session (Compared to Typical Discussion Sections) {N = 25) 



Much more 
effective 

More 

effective 

Similar 

Less 

effective 

Much less 
effective 

Learning course material 

3 

12 

7 

2 

1 

Interaction 

5 

11 

7 

2 

0 

Engagement 

8 

15 

0 

2 

0 


Table 3 (next page) shows perceived impacts on specific aspects of the 
course. Surprisingly, about one third of students reported the computer- 
based discussion section had a big or very big effect on course outcomes 
(homework, exams, grades). Students found the computer-based dis- 
cussion section having a big or very big effect on completing homework 
assignments. This is unsurprising as the problems covered in the discus- 
sion section were taken from the textbook or closely modified homework 
problems. The perceptions that the computer-based discussion section 
helped with performance on exams and the course was encouraging, as it 
suggests that about a quarter of the students perceived the experience as 
highly useful. However, as mentioned earlier, no statistical relation was 
found between participation and course outcomes. 

Even more surprising was that the experience resulted in some 
students reporting a big or very big impact on non-cognitive outcomes 
such as attitudes toward engineering in general and in making new friends/ 
networking. These are interesting findings given the variability of treat- 
ment effects (Table 1, page 8). Further, these results suggest that there 
may be side-effects to this type of instructional delivery. That is, while the 
original focus was to improve learning, about a quarter of the students 
apparently perceived attitudinal benefits as well. 
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Table 3: Students' Perception of the Impact the Computer-Based Session 

had on Different Course Outcomes 



n 

Very big 
effect 

Big 

effect 

Some 

effect 

Little 

effect 

No 

effect 

Learning of specific analysis 
techniques 

25 

3 

3 

8 

7 

4 

Development of circuit analysis 
problem solving skills 

25 

2 

1 

11 

7 

4 

Completing homework assignments 

24 

5 

7 

7 

2 

3 

Performance on exams 

24 

2 

7 

9 

2 

4 

Performance on the course 

24 

2 

4 

1 1 

4 

3 

Your improvement on attitudes 
toward electrical engineering 

24 

2 

3 

13 

5 

1 

Your attitudes toward 

EEl 0/EEl 1 5A/BREES in particular 

24 

0 

4 

8 

10 

3 

Your attitudes toward engineering 
in general 

24 

3 

4 

10 

6 

1 

Your making new friends/networking 

24 

6 

5 

7 

5 

1 


We interpret these results as students perceiving the computer-hased 
discussion section as henign at worst, and in many cases more effective 
than typical discussion sections. Students’ perception of effects on most of 
the course and attitudinal outcomes were generally positive. In particular, 
students appeared to perceive engagement as the most powerful impact of 
the computer-hased discussion section. In the following section, student 
interviews provide insight underlying the survey responses. 


Analyses of Student and Instructor Interviews 

Perceived learning of course material. The computer-hased discussion 
section required students to understand every step and the professor to 
explain and clarify those steps. In general, the analyses of student inter- 
views suggest that students felt more in control of the material and under- 
stood it more thoroughly. While students reported that the discussions 
were helpful because it forced them to solve the problem, the technology 
alone did not constitute good instruction. One student, “the fact that he 
[the professor] takes the problem and solved it on the board and explained 
it in detail. That’s the most useful thing.” 

But this mode of instruction was not helpful for all students, particu- 
larly for those who were able to solve the problem quickly. One survey 
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respondent commented; “The process of learning seems to be slowed down 
a little too much. Though the discussion can be changed to fit our learning 
needs, every example takes too long. This makes the class a little boring.” 

Perceived Interaction 

One of the most important dimensions reported by students as a 
barrier for their learning was related to their social inhibitions. Nearly 
every student we interviewed had said that he or she felt intimidated in 
a large classroom and in the presence of a professor to raise their hands 
and ask to clarify incomprehensible material. Students expressed their 
concerns that professors would consider their questions to be stupid or 
inappropriate, and that professors would remember them as students who 
asked stupid questions: 

For example... I'm hesitant to ask questions during class and since this is 
anonymous. . .the professor won't point anyone out or anything so he'll just 
be able to answer a lot of questions, and then you'll see a lot of students 
making common errors. So he'll be able to answer those specific ones. 

In the computer-based discussion section, student responses were 
anonymous and an individual could correspond with the professor directly. 
Students felt that anonymity was very important to participation, and 
participation increased their level of understanding of the material. One 
survey respondent commented about the anonymity: 

I think that maintaining anonymity is very crucial in the interaction aspect 
of the discussion. Many, including myself may feel a little embarrassed 
asking a "dumb"question but with this method, I don't feel that people will 
hesitate to ask those questions. 

While many students cited anonymity as important, not all students 
felt this way. A small number of students reported that it simply would be 
easier to just raise their hands and ask questions. 
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Perceived Engagement 

A seemingly obvious activity that should be occurring in a circuit anal- 
yses discussion section - computer or no computer - is problem solving. 
However, students’ interview responses suggest this is typically not the 
case in non-computer sections. Thus, one important benefit of the com- 
puter-based discussion section was that it required students to solve 
circuit problems. Each student had his or her own laptop and was required 
to solve circuit analysis problems. When students were asked about their 
level of engagement, all interviewees said engagement was much higher 
in the computer-based discussion section compared to typical discussion 
sections. For example, one student said: 

Definitely, ten times more engaged. [In a typical discussion section] 

You don't have to engage in a lecture or whatever's going on in class. . . 

You can just sit there and do something else, do homework or whatever. 

But in that [computer-based] class, we didn't have any other papers 
around, we all had a terminal, and everyone else in the class was 
participating, and doing problems. . .and you couldn't do anything 
else but participate. So definitely more participation, definitely more. 

Interestingly, the instructor also reported that his students were 
much more engaged than the typical section. When we asked him about 
his students’ level of engagement he said that it was much more focused. 

In a normal group, I would lose some people. They would fall asleep. 

I think if they're not getting something, they'll just say,"l'm going home 
and study this." And you'd lose 10% of the people at any given time. This 
session was not, at all. Everyone was very engaged. Everyone was at full 
attention. When you work on a problem, everyone would be watching 
the problem solution very intently. When the problem was assigned, 
everyone would be really working on the problem. That was another 
very interesting point. 
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Students' Perception of Non-cognitive Outcomes 

Finally, one of the most unexpected findings emerging from the 
student interviews was the impact the experience had on students’ non- 
cognitive outcomes. That is, the issue of a caring professor emerged - one 
who creates an open and non-intimidating atmosphere to ask questions. 
All participants said that the instructor’s attitude had heen one of the 
main reasons that helped them learn. The instructor taking time to teach 
and to explain fundamental material was one of the most important 
factors cited as leading to students’ increase in the level of engagement 
and understanding. It is essential to add that we did not prompt students 
for these comments on the instructor - all interviewees provided unsolic- 
ited feedback on the instructor. 

Professor [X], he helped us out with that it felt a little bit better getting it 
from a professor than aTA. I don't know why but it just did. But he was real 
kind, kind at answering everything pretty much. He was able to answer. . . 
all my questions. I was happy with this computer thing. 

Monitoring Students' Performance 

One of the most important benefits of the computer-based discussion 
section was the capability for the instructor to simultaneously observe 
students’ problem-solving responses in real-time. Contrast this situa- 
tion with the typical lecture class, where the instructor has little feedback 
about whether the students are comprehending the material. Thus, the 
instructor could determine how many students did or did not understand 
a concept and could address the situation immediately. In the instructor’s 
words: 

What happens is that I get a chance to see exactly what people were 
doing, and if there was an error... what I would then do is spend discussion 
time speaking about that particular error, and instructing them about 
that particular problem, and I found that to be immensely more effective 
than the alternative. Now, in EE1 0. . . we were able to provide a. . .set of 
problems ranging from relatively simple to complex. And, at the beginning 
of the session, the success rate of the class might only be like one 
third... correctly solving the simple problems. But by the end of the class 
session, virtually the entire class was. . .working through the most difficult 
problems. So, the progress over a one-hour period was general. I would say, 
it could even exceed one-on-one office hours. Because, even at the one- 
on-one office hours, I typically don't ask a student to solve a problem. I'll do 
it for them, and ask them about it or they'll ask me how we move from step 
to step. And that's much better than a large group in terms of interaction 
and quality of interaction but even there we're not seeing what exactly the 
student does on their own. Because it's somewhat confrontational to ask 
them to solve a problem right there on the spot - they're nervous. 
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Interestingly, students also reported the benefits of the instructor’s 
capability to observe students’ responses. They reported the instruction 
being more targeted and individualized because the instructor was able to 
focus on what students were not getting correct. For example, in response 
to the question of what he had gained from the computer-based discussion 
section, one student remarked: 

I can say the most I gained was a little bit better problem-solving skill as far 
as analysis was concerned. I think that professors sort of looked at mistakes 
we were making along the way. . .So to get the idea at that exact moment 
where our mistakes [are] coming from and so he was able to more specify 
what he needed to teach, what the emphasis of learning should be on. 

Discussion 

The most effective form of assessment is one that is continuous, that 
occurs as close as possible to the scene of the action in teaching and learning 
(the classroom), and that provides diagnostic feedback to both teachers 
and students - to teachers on how they can improve their teaching, to 
students on how they can improve their learning (Cross, 1988, p. 3). 

Our assessment perspective embraces this idea. This study directly 
attempted to address the issue of improving student learning via an online 
formative assessment that provided continuous feedback to instructors 
about students’ learning progress, who could then provide feedback to 
students about their performance and provide remediation as needed. 
Effective feedback to students is consistently cited as one of the most 
important components of an effective learning environment, and it has 
been found to increase not only student learning, but also confidence and 
attitudes (e.g., Angelo and Cross, 1993; Black and Wiliam, 1998; Bloom, 
1984; Chickering and Gamson, 1987; Cross, 1988, 1998; McKeachie, 1998; 
Tinto, 2002; Twigg, 1999). 

Students overwhelmingly reported positive outcomes for the com- 
puter-based discussion section. When asked to compare their experience 
with traditional discussion sections, students indicated the computer- 
based discussion section had superior engagement with the content, 
had increased positive attitudes toward the course and engineering, and 
had increased learning of the material. Students felt they shifted from a 
passive role to a much more active and engaged role and felt they learned 
the material more thoroughly. Because students were required to solve 
problems step-by-step (vs. typical instruction where material was intro- 
duced broadly and students left on their own), students had to ask ques- 
tions and “experience the equations.” One of the most interesting findings 
was the bolstering of students’ confidence in solving circuit problems 
and affective outcomes. The real-time feedback from the instructor, often 
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individualized, was cited as critical not only to increasing content under- 
standing but also because students felt they were receiving personal atten- 
tion. Similarly, anonymity was critical because students felt comfortable 
asking “dumb” questions. 

Limitations and Future Directions 

The main limitation of this study was lack of an experimental design. 
This was intentional as we were investigating the feasibility and utility of 
the approach. The disadvantage of this tactic is that the findings of this 
study, suggestive as students’ self-reports are, do not confirm the exis- 
tence of a clear learning impact. Other limitations include a small sample 
size, a self-selected sample, and testing in only three courses from the 
same school of engineering. Also, while the instructional format appears 
useful for problem solving, it is unclear whether the format would be 
appropriate for non-problem solving-related topics. However, given the 
provocative findings of the current study, a clear next step is to contrast 
students’ performance on various course outcomes between students 
randomly assigned to the computer-based discussion section and a control 
group, controlling for instructor effect. Such a study would need to be of 
sufficient duration to allow the intervention to have an impact. 

The second line of future work is developing methods to allow scaling 
of the approach to large lecture classes. We have tested the system in 
classes with as many as 60 students (not part of this study); while still 
informative, attempting to observe all students’ responses and to attend 
to individual questions or problems quickly becomes overwhelming. Thus, 
we are exploring techniques that blend automated reasoning approaches 
(e.g., Bayesian networks) and other formative assessment techniques (e.g., 
self-ratings of level of understanding; muddiest point) to allow students to 
engage and participate, while also providing a mechanism that will allow 
partial filtering of responses and automated detection and summarization 
of student responses. 

The lecture format is likely to remain a large part of undergraduate 
instruction for the foreseeable future, and consequently its impact on 
student learning will be pervasive. Our work is one step toward developing 
online formative assessment methods intended to increase the level of 
engagement and interaction in the traditional lecture format. 
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